Ge/19 #20

gegnew · 2019-11-01T22:30:26Z

Also lots of formatting.

gegnew · 2019-11-02T16:52:54Z

I need to resolve the namespace of the by_name function, for two reasons:

Need to have an accessible cache_clear method
Need to make sure that duplicate names representing different objects do not conflict (i.e. the example in the R toolkit)

gegnew · 2019-11-04T12:22:26Z

The LRU cache (which currently has a 2048 item limit, which can be made unlimited) works for different objects (like files or gates) inside different experiments with the same name. There are accessors to the cache_clear and cache_info methods in the base module.

JobLeonard

So aside from a few PEP8 related nits, I'm wondering if it wouldn make the code easier to use if we take that massive list of optional parameter options for gate creation:

    label=[], gid=None, locked=False,
    parent_population_id=None, parent_population=None,
    tailored_per_file=False, fcs_file_id=None,
    fcs_file=None, create_population=True

... and make a little class for that (using __slots__ of course, for better performance)

class GateOptions:
    __slots__ ="label", "gid", "locked", "parent_population_id",
            "parent_population", "tailored_per_file", "fcs_file_id",
            "fcs_file", "create_population"

    def __init__(self, label=None, gid=None, locked=False,
                parent_population_id=None, parent_population=None,
                tailored_per_file=False, fcs_file_id=None,
                fcs_file=None, create_population=True):
        if label is None:
            label = []
        self.label = label
        self.gid = gid
        self.locked = locked
        self.parent_population_id = parent_population_id
        self.parent_population = parent_population
        self.tailored_per_file = tailored_per_file
        self.fcs_file_id = fcs_file_id
        self.fcs_file = fcs_file
        self.create_population = create_population

.. then you can replace those parameters with something like gateOpts=None in the parameters, and:

if gateOpts is None:
    gateOpts = GateOptions()

... to initialize the default values in the function (which you can then pass on to subsequent calls as well).

The downside would be that when the user would want to pass these options, they would have to write:

create_range_gate(experiment_id, x_channel, name, x1, x2, y, GateOptions(...))

instead of:

create_range_gate(experiment_id, x_channel, name, x1, x2, y, ...)

OTOH, maybe they'll just end up creating a few default options objects and re-use them.

JobLeonard · 2019-11-04T15:09:10Z

README.md

+##Developer Notes
+- `id` is a python builtin, which causes some confusion. We use `_id` to indicate
+the ID of an API object, but the `attrs` package does not accept leading
+underscores (i.e. an `_id = attr.ib()` in a class is treated by `attrs` as the
+string "id"). Practically, this means:
+    - pass `_id` to functions that take an ID as an argument.
+    - pass `id` to `attrs` classes when instantiating them.
+    - pass `properties` to `attrs` classes when instantiating them.


In Python, leading underscores are used to signal private properties to programmers. The Naming Styles section of PEP 8 says this:

_single_leading_underscore: weak "internal use" indicator. E.g. from M import * does not import objects whose names start with an underscore.

single_trailing_underscore_: used by convention to avoid conflicts with Python keyword, e.g.

Tkinter.Toplevel(master, class_='ClassName')

Let's replace _id with id_ and see if it also fixes the attrs interop?

I think that's gonna be a messy hassle to convert back and forth since our API uses _id... If the only known bug/limitation from using _xxx is due to attrs, I'd consider dropping that dependency first.

Well, keep in mind that these kind of interop issues are likely happen with other Python libraries as well, because the entire Python ecosystem expects this convention.

So from that angle the argument becomes one of weighing how important it is to be consistent between Python/R/JavaScript, and within each language.

Are Python users likely to deal with the other APIs, or more likely to stick to the Python environment?

I wish MongoDB didn't use _id...

It sounds like a single leading underscore is only a convention and hint. I'm curious why attrs breaks. I skimmed the source and searched the docs and only find this bit (ctrl-f "leading underscores") http://www.attrs.org/en/stable/examples.html or (ditto) http://www.attrs.org/en/stable/api.html. Also python-attrs/attrs#391

Is there at least some way in Python to automatically do the conversion so we never have code like req_body = {_id: self.id_}? e.g. an _id getter?

All of our API documentation uses _id (which is factual). To the extent possible, I'd like to keep consistency.

I guess we already have marshalling for our API's camelCase to Python's snake_case, so maybe I shouldn't worry.

It sounds like a single leading underscore is only a convention and hint.

Well, to quote PEP8 again: "E.g. from M import * does not import objects whose names start with an underscore", and import is a language-level keyword so it's more than a convention I'm afraid. It's more like a language feature that is not enforced by the language, because dynamic typing and monkey patching and whatnot (no, I'm not a fan either)

I'm curious why attrs breaks. I skimmed the source and searched the docs and only find this bit (ctrl-f "leading underscores") http://www.attrs.org/en/stable/examples.html or (ditto) http://www.attrs.org/en/stable/api.html. Also python-attrs/attrs#391

If I read those docs correctly, the only real problem here is that keyword arguments don't accept leading underscores. So calling SomeGatingClass(_id=1234) would fail, but SomeGatingClass(id=1234) would work fine.

I suppose attrs does this because the very idea of private properties means that they Should Not Be Parameter Names (that is, I'm passing some property x that is stored privately as _x).

Is there at least some way in Python to automatically do the conversion so we never have code like req_body = {_id: self.id_}? e.g. an _id getter?

I think @property decorator is your friend, although I am not entirely sure if it covers your use-case:

@property _id(self): return self.id_ @_id.setter def _id(self, value): self.id_ = value @_id.deleter def _id(self): del self.id_

I have changed the README to be a little more clear: basically, a user should always pass _id and never worry about this naming convention otherwise. Internally, everything should be named _id, but it is possible that you might have to pass id as an init arg somewhere. The only case I can find of this right now is actally the _properties arg in loader.Loader().make_class. There, properties is passed to a given <classname>, which contains a _properties attribute.

Otherwise, regarding this branch, I understand from this conversation that we should:

keep _id as it is

not use a GateOptions class for gate options

In addition, I have made all the main data classes slots classes in the next branch.

Sounds good to me but it's Zach's final call.

Nice @ slots, if I understand the use-case for this lib correctly I doubt we'll be crunching through hundreds of thousands of instances, but keeping things lean still feels good :)

cellengine/Gates/gate_util.py

zbjornson · 2019-11-04T18:54:09Z

The downside would be that when the user would want to pass these options, they would have to write:

create_range_gate(experiment_id, x_channel, name, x1, x2, y, GateOptions(...))

I think I'd avoid GateOptions for that reason. Could you have a function like initializeGateDefaults(self) though, as far as deduping default init?

JobLeonard · 2019-11-05T17:01:12Z

I think I'd avoid GateOptions for that reason. Could you have a function like initializeGateDefaults(self) though, as far as deduping default init?

These are arguments in top-level functions like create_polygon_gate, create_range_gate and so on, not classes, so that won't work. And I think Gerrit already refactored out the common code into common_gate_create (tangent: I just noticed that all of these functions follow the naming convention create_xxx_gate, except common_gate_create. Maybe make that create_common_gate for the sake of consistency? Or are those names also modelled after the R toolkit?)

If I understand the code correctly there's one generic Gate class, plus a bunch of functions that help with initializing a Gate object with the specific settings for each gate type.

These create_xxx_gate functions all have default values though, so they don't require users to pass all of them, and Python lets you use keyword arguments to skip the parameters you don't care about IIRC - so you could do create_quadrant_gate(experiment_id, x_channel, y_channel, name, x, y, fcs_file_id=<somefileid>) and skip all the parameters before fcs_file_id. So might actually be ok to use in practice I guess - but 16 parameter functions just make me do a double take.

… methods Population resources complete, child_gate model func written partial method for creating complex gates added complex population creator; needs tests create_complex_population tested split data objects from their methods methods + tests creating child gates added update and delete methods

gegnew requested review from JobLeonard and zbjornson November 1, 2019 22:30

gegnew force-pushed the ge/19 branch from 5ca3f0d to bdd4d79 Compare November 4, 2019 12:20

gegnew marked this pull request as ready for review November 4, 2019 12:20

gegnew force-pushed the ge/19 branch 2 times, most recently from d5fcb84 to 58d32e7 Compare November 4, 2019 15:37

JobLeonard requested changes Nov 4, 2019

View reviewed changes

gegnew force-pushed the ge/19 branch 2 times, most recently from 7209121 to 9eb8c57 Compare November 6, 2019 15:43

gegnew force-pushed the ge/19 branch from 9eb8c57 to 2cf5261 Compare November 26, 2019 12:25

gegnew added 4 commits December 1, 2019 16:49

new factory for gate creation

d1d3ec5

change create_common_gate name

52cc02a

change create_common_gate name

fd2cad9

no idea whats happening

6cd1e1c

gegnew force-pushed the ge/19 branch from 2cf5261 to 6cd1e1c Compare December 1, 2019 19:52

gegnew closed this Dec 1, 2019

gegnew deleted the ge/19 branch December 1, 2019 20:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Ge/19 #20

Ge/19 #20

Uh oh!

gegnew commented Nov 1, 2019

Uh oh!

gegnew commented Nov 2, 2019

Uh oh!

gegnew commented Nov 4, 2019

Uh oh!

JobLeonard left a comment •

edited

Loading

Uh oh!

JobLeonard Nov 4, 2019

Uh oh!

zbjornson Nov 4, 2019

Uh oh!

JobLeonard Nov 4, 2019

Uh oh!

zbjornson Nov 4, 2019

Uh oh!

zbjornson Nov 4, 2019

Uh oh!

JobLeonard Nov 5, 2019 •

edited

Loading

Uh oh!

gegnew Nov 6, 2019 •

edited

Loading

Uh oh!

JobLeonard Nov 6, 2019

Uh oh!

Uh oh!

Uh oh!

zbjornson commented Nov 4, 2019

Uh oh!

JobLeonard commented Nov 5, 2019

Uh oh!

Uh oh!

Ge/19 #20

Ge/19 #20

Uh oh!

Conversation

gegnew commented Nov 1, 2019

Uh oh!

gegnew commented Nov 2, 2019

Uh oh!

gegnew commented Nov 4, 2019

Uh oh!

JobLeonard left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JobLeonard Nov 4, 2019

Choose a reason for hiding this comment

Uh oh!

zbjornson Nov 4, 2019

Choose a reason for hiding this comment

Uh oh!

JobLeonard Nov 4, 2019

Choose a reason for hiding this comment

Uh oh!

zbjornson Nov 4, 2019

Choose a reason for hiding this comment

Uh oh!

zbjornson Nov 4, 2019

Choose a reason for hiding this comment

Uh oh!

JobLeonard Nov 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gegnew Nov 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JobLeonard Nov 6, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zbjornson commented Nov 4, 2019

Uh oh!

JobLeonard commented Nov 5, 2019

Uh oh!

Uh oh!

JobLeonard left a comment •

edited

Loading

JobLeonard Nov 5, 2019 •

edited

Loading

gegnew Nov 6, 2019 •

edited

Loading